Computer system having an apportionable data bus and daisy chained memory chips

ABSTRACT

A memory system having a data bus coupling a memory controller and a memory. The data bus has a number of data bus bits. The data bus is programmably apportioned to a first portion dedicated to transmitting data from the memory controller to the memory and a second portion dedicated to transmitting data from the memory to the memory controller. The apportionment can be assigned by suitable connection of pins on a memory chip in the memory and the memory controller to logical values. Alternatively, the apportionment can be scanned into the memory controller and the memory at bring up time. In another alternative, the apportionment can be changed by suspending data transfer and dynamically changing the sizes of the first portion and the second portion.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to:

U.S. application Ser. No. 11/459,956, filed on Jul. 26, 2006, entitledDaisy Chained Memory System.

U.S. application Ser. No. 11/459,957, filed on Jul. 26, 2006, entitledMemory System Having Self Timed Daisy Chained Memory Chips.

U.S. application Ser. No. 11/459,969, filed on Jul. 26, 2006, entitledCarrier Having Daisy Chained Memory Chips.

U.S. application Ser. No. 11/459,983, filed on Jul. 26, 2006, entitledCarrier Having Daisy Chain of Self Timed Memory Chips.

U.S. application Ser. No. 11/459,994, filed on Jul. 26, 2006, entitledDaisy Chainable Memory Chip.

U.S. application Ser. No. 11/459,997, filed on Jul. 26, 2006, entitledDaisy Chainable Self Timed Memory Chip.

U.S. application Ser. No. 11/459,974, filed on Jul. 26, 2006, entitledComputer System Having Daisy Chained Memory Chips.

U.S. application Ser. No. 11/459,968, filed on Jul. 26, 2006, entitledComputer System Having Daisy Chained Self Timed Memory Chips.

U.S. application Ser. No. 11/459,966, filed on Jul. 26, 2006, entitledMemory Controller For Daisy Chained Memory Chips.

U.S. application Ser. No. 11/459,961, filed on Jul. 26, 2006, entitledMemory Controller For Daisy Chained Self Timed Memory Chips.

U.S. application Ser. No. 11/459,943, filed on Jul. 26, 2006, entitledMemory Chip Having an Apportionable Data Bus.

U.S. application Ser. No. 11/459,947, filed on Jul. 26, 2006, entitledSelf Timed Memory Chip Having an Apportionable Data Bus.

U.S. application Ser. No. 11/459,959, filed on Jul. 26, 2006, entitledMemory System Having an Apportionable Data Bus and Daisy Chained MemoryChips

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to computer systems, memory systems,and memory interconnections in electronic systems. More particularly,the invention relates to high speed transmission of data in a computersystem.

2. Description of the Related Art

Modern computer systems typically are configured with a large amount ofmemory in order to provide data and instructions to one or moreprocessors in the computer systems.

Historically, processor speeds have increased more rapidly than memoryaccess times to large portions of memory, in particular, DRAM memory(Dynamic Random Access Memory). Memory hierarchies have been constructedto reduce the performance mismatches between processors and memory. Forexample, most modern processors are constructed having an L1 (level 1)cache, constructed of SRAM (Static Random Access Memory) on a processorsemiconductor chip. L1 cache is very fast, providing reads and writes inonly one, or several cycles of the processor. However, L1 caches, whilevery fast, are also quite small, perhaps 64 KB (Kilobytes) to 256 KB. AnL2 (Level 2) cache is often also implemented on the processor chip. L2cache is typically also constructed of SRAM design, although someprocessors utilize DRAM design. The L2 cache is typically several timeslarger in number of bytes than the L1 cache, but is slower to read orwrite. Some modern processor chips also contain an L3 (Level 3) cache.L3 cache is capable of holding several times more data than the L2cache. L3 cache is sometimes constructed with DRAM design. L3 cache insome computer systems is implemented on a separate chip or chips fromthe processor, and is coupled to the processor with wiring on a printedwiring board (PWB) or a multi chip module (MCM). Main memory of thecomputer system is typically large, often many GB (gigabytes) and istypically implemented in DRAM.

Main memory is typically coupled to a processor with a memorycontroller. The memory controller receives load (read) commands andstore (write) commands from the processor and services those commands,reading data from main memory or writing data to main memory. Typically,the memory controller has one or more queues (e.g., read queues andwrite queues). The read queues and write queues buffer information(e.g., commands, addresses, data) so that the processor can havemultiple read and/or write requests in progress at a given time.

In various implementations, signaling between the memory controller andthe memory chips comprise multidrop connections. That is, a pin on thememory controller connects directly to a plurality of memory chip pins(e.g., DRAM chip input or output or common I/O connection) It will beunderstood that typically one memory chip is placed on one module, sothe connection to a particular memory chip includes a module pin plusthe chip pin. Occasionally, several memory chips are placed on a singlemodule which creates multiple drops even on a single module.

Another approach uses point to point interconnections between the memorycontroller and a buffer chip, the buffer chip being associated with anumber of memory chips and accessing (writing/reading) to/from thoseassociated chips when the buffer chip receives an address on the pointto point interconnect from the memory controller. If the addressreceived does not address the memory chips associated with the bufferchip, the buffer chip re-drives the command/address, and perhaps data,to another buffer chip.

FIG. 1 illustrates such a prior art memory structure. Memory controller12 is coupled to a first point to point interconnection 18A, comprising“M” bits to a first buffer chip 20A. First point to pointinterconnection 18A carries address and command information. Memorycontroller 12 is coupled to a second point to point interconnection 19A,comprising “N” bits, to the first buffer chip 20A. Buffer chip 20A ismounted on a carrier 16A. Also shown mounted on carrier 16A are eightmemory chips 14. Buffer chip 20A, as described above, receives addressand command information on first point to point interconnect 18A. Ifbuffer chip 20A determines that the address received addresses data inan address space of carrier 16A, buffer chip 20A drives address andcontrol information on multidrop interconnection 21A. Data is typicallysent on multiple, point to point interconnections between buffer chip20A and memory chips 14 as shown on point to point connections 22 (foursuch point to point connections are referenced with numeral 22, forsimplicity, others are not explicitly referenced). If, however, bufferchip 20A determines that the address received on first point to pointinterconnect 18A does not address the memory space of carrier 16A,buffer chip 20A retransmits the address and command on point to pointinterconnect 18B to a second buffer chip 20B. Buffer chip 20B is mountedon carrier 16B and is coupled to memory chips 14 on carrier 16B. Ifbuffer chip 20B determines that the address is not for an address spaceof carrier 16B, buffer chip 20B further re-drives the address andcommand on point to point interconnect 18C to a third buffer chip (notshown). If buffer chip 20B determines that the address is for the memoryaddress space of carrier 16B, buffer chip 20B drives address and controlinformation on multidrop interconnection 21B.

Data is sent, as described above, on point to point interconnections 22between buffer chip 20B and memory chips 14 on carrier 16B (as before,four point to point connections 22 shown referenced). Thus, the addressand command data is “daisy-chained” from one buffer chip 20 to another,with the appropriate buffer chip reading or writing data from/onto pointto point interconnects 19 (shown as 19A-19C in FIG. 1). A problem withthis approach is that buffer chips are required. Buffer chip 20 takes uparea on carrier 16, and dissipates power. In electronic packaging andsystem design, area and power consumption are typically desired to beminimized. Buffer chips also add cost to a memory system. Yet anotherproblem in this implementation is that a first period of time (one ormore cycles) is used to drive the address and command to a buffer chipand a second period of time (one or more cycles) is then used to drivethe address on a carrier (e.g., carrier 16). Driving signals on carrierinterconnect, such as copper wiring on a printed wiring board (PWB)requires significant area on the buffer chip for the off chip driver,and associated ESD (electrostatic discharge) circuitry. Ensuring thatthe chip—module—carrier—module—chip path is operational, and providingfor diagnosis of faulty signaling paths, also often requires that someor all pins be driven by a common I/O circuit that can both drive andreceive, thus increasing the size and complexity of the circuitry thatdrives (or receives).

Current memory systems comprise one or more data busses that carry writedata from the memory controller to the memory chips for storing inarrays in the memory chips. The data busses also carry read data to thememory controller from the memory chips. The data busses of currentmemory systems have a fixed bandwidth for write data versus read data.Different workloads in computer systems require different readbandwidths versus write bandwidths. For example, a numerically intensiveworkload is best served with a read bandwidth approximately the same asa write bandwidth. In contrast, a commercial database workload requiresmore read bandwidth than write bandwidth. Current memory systems areunable to accommodate different read versus write bandwidths other thanby running at least some workloads inefficiently.

Therefore, there is a need for further improvement in a fast andefficient memory system.

SUMMARY OF THE INVENTION

The present invention provides a computer system having an apportionabledata bus. In an exemplary description, a memory system having a memorycontroller and a memory is described. At least one data bus connects thememory controller to the memory. The data bus has “N” data bus bits. The“N” data bus bits are programmably apportioned to a first portiondedicated to carrying write data from the memory controller to thememory, and to a second portion dedicated to carrying read data from thememory to the memory controller.

In an embodiment, a processor transmits an apportionment to the memorycontroller, which forwards the apportionment to memory chips in thememory. The memory controller causes transmission of read data and writedata to be suspended during changes in apportionment in order thatdrive/receive conflicts are avoided on the data bus.

In an embodiment, I/O pins on the memory controller are connected to oneor more sources of logical value that, when powered, provide a value forthe apportionment. The memory controller transmits the apportionment tothe memory chips in the memory.

In an embodiment I/O pins on the memory controller and I/O pins onmemory chips in the memory are connect to one or more sources of logicalvalue that, when powered, provide a value for the apportionment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is prior art drawing of a memory controller and two memorycarriers, each memory carrier having a buffer chip and a plurality ofmemory chips.

FIG. 2 is a high level block diagram of a computer system embodying thepresent invention

FIG. 3A is a block diagram of a memory system having a memory controllerand two memory carriers according to an embodiment of the invention.

FIG. 3B is a block diagram of a memory system having a memory controllerand two memory carriers according to an alternative embodiment of theinvention.

FIG. 4 is a block diagram of a memory chip according to an embodiment ofthe invention.

FIG. 5A shows a representative address/command word and its subportions.

FIG. 5B shows an alternate embodiment of an address/command word and itssubportions.

FIG. 5C shows another alternate embodiment of an address/command wordand its subportions.

FIG. 6A shows a block diagram of an address/command block, with detailstherein.

FIG. 6B shows a block diagram of a memory controller and four daisychained memory chips with bus referencing to be used in FIG. 6C.

FIG. 6C shows an address/command word that addresses a fourth memorychip in FIG. 6B, and describes how a chip ID value is shifted as theaddress command word passes through the daisy chained memory chips.

FIG. 6D shows a block diagram of an address/command block according toan alternate embodiment of the invention.

FIG. 7A is a block diagram of a data logic block in a memory chip.

FIG. 7B is an exemplary data word received or sent by the data logicblock in the memory chip.

FIG. 7C is an alternative exemplary data word that includes a chip IDfield.

FIG. 7D is a block diagram of an alternate embodiment of a data logicblock in a memory chip.

FIG. 7E is a block diagram of an alternative data logic block suitablefor programmable apportionment of a data bus.

FIG. 7F is a block diagram showing details of how programmableapportionment of a data bus is implemented.

FIG. 8 is a block diagram illustrating an alternative interconnectionembodiment of the invention.

FIG. 9A is a block diagram similar to that shown in FIG. 8, but usingself timed memory chips.

FIG. 9B is a block diagram of the memory chip of FIG. 9A, showingfurther details of how an array on the memory chip is self timed.

FIG. 10 is a block diagram illustrating components of a self time blockshown in FIG. 9B.

FIG. 11A is a block diagram illustrating a first embodiment of an arraytiming control block shown in FIG. 10.

FIG. 11B is a block diagram illustrating a second embodiment of thearray timing control block shown in FIG. 10.

FIG. 12 is a block diagram showing alternate signal routings of a databus chain is routed through a daisy chain of memory chips.

FIG. 13 is a block diagram of a memory controller according toembodiments of the invention.

FIG. 14 is a flowchart of a method embodiment of the invention.

FIG. 15 is a flowchart of a method performed by a memory chip accordingto an embodiment of the invention.

FIG. 16 is a more detailed flowchart of a method performed by a memorychip according to an embodiment of the invention.

FIG. 17 is a flowchart of a self timing method according to anembodiment of the invention.

FIG. 18A illustrates an address/command word in which the chip ID is aportion of the address. No packet ID is implemented.

FIG. 18B illustrates a data word having no packet ID.

FIG. 18C illustrates a memory controller that drives address/commandwords timed to ensure that no collisions occur on the data bus chain.

FIG. 19 is a flow chart of a method that provides programmableapportionment of a data bus.

FIG. 20 is a flow chart illustrating return of data in a different orderthan corresponding address/command words.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following detailed description of the preferred embodiments,reference is made to the accompanying drawings, which form a parthereof, and within which are shown by way of illustration specificembodiments by which the invention may be practiced. It is to beunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the invention.

The present invention provides a computer system having an apportionabledata bus. In an exemplary description, a memory system having a memorycontroller and a memory is described. At least one data bus connects thememory controller to the memory. The data bus has “N” data bus bits. The“N” data bus bits are programmably apportioned to a first portiondedicated to carrying write data from the memory controller to thememory, and to a second portion dedicated to carrying read data from thememory to the memory controller.

In an embodiment, a processor transmits an apportionment to the memorycontroller, which forwards the apportionment to memory chips in thememory. The memory controller causes transmission of read data and writedata to be suspended during changes in apportionment in order thatdrive/receive conflicts are avoided on the data bus.

In an embodiment, I/O pins on the memory controller are connected to oneor more sources of logical value that, when powered, provide a value forthe apportionment. The memory controller transmits the apportionment tothe memory chips in the memory.

In an embodiment I/O pins on the memory controller and I/O pins onmemory chips in the memory are connect to one or more sources of logicalvalue that, when powered, provide a value for the apportionment.

Turning now to FIG. 2, an exemplary computer system 250 having anembodiment of the present invention is shown in block form sufficientfor an understanding of computer system 250. A processor 200 is coupledby bus 223 to a memory controller 52 in a memory system 270. Processor200 issues fetch and store commands via bus 223 to memory controller 52,including address information of what memory locations to store to orfetch from. Memory controller 52 processes the fetch and store commands,doing any required logical to physical address translations, determiningwhich chip in a daisy chain of memory chips is to be used to handle thecommands, and fetching/storing data from/to a memory 210 via bus 224.Embodiments of bus 224 will be explained in detail later. Bus 224 is oneor more separate busses that send address/command information to memorychips on point to point interconnections and send or receive data fromthe memory chips on point to point interconnections.

Processor 200 is also coupled to an I/O controller 53 via bus 222. I/Ocontroller 53 serves as a controller for data going to and coming fromstorage devices such as CDROMs, hard disks, magnetic tapes, and thelike; user interface devices such as mice, keyboards, and the like; andnetwork devices such as modems, Ethernet interfaces and the like.Typically, I/O controller 53 is coupled via I/O bus 225 to a storagecontroller 201 which is further coupled to storage devices 204; to auser interface controller 202 which is further coupled to user interfacedevices 205; and network controller 203 which is further coupled tonetwork devices 206. Computer system 250 is exemplary only, and it willbe understood that often computer systems comprise a plurality ofprocessors, and different computer systems are coupled to storagedevices 204, user interface devices 205, and network devices 206 invarious alternative interconnection methods.

FIG. 3A shows memory system 270 with memory controller 52 coupled by bus224 to memory 210, with memory 210 shown in more detail according to anembodiment of the invention. Bus 224 comprises address/command bus 58and data bus 59. In an embodiment, address/command bus 58 and data bus59 interconnect memory controller 52 with daisy chained memory chips 54.For purposes of identification of particular address/command busses 58and data busses 59, letters are appended to particular address/commandbusses 58 and data busses 59. Carriers 56 (shown referenced as carrier56A and carrier 56B) have memory chips 54 attached. Memory chips 54A-54Jare attached to carrier 56A; memory chips 54M-54V are attached tocarrier 56B. A first daisy chain of memory chips 47A, in FIG. 3A,comprises memory chips 54A, 54M, and any other memory chips 54 seriallycoupled with memory chips 54A, 54M. Similarly, memory chips 54J, 54V,and any other memory chips 54 serially coupled with memory chips 54A,54M are a second daisy chain of memory chips 47B, in FIG. 3A. As shown,other “horizontal rows” of memory chips make up other daisy chains ofmemory chips 47. Generically, a daisy chain of memory chips is denotedby reference numeral “47”, suffixed as needed to describe particulardaisy chains of memory chips 47. Daisy chains of memory chips 47A and47B are in dotted boxes having an open end to illustrate possibleadditional memory chips 54 in each daisy chain of memory chips 47.

In the embodiment depicted in FIG. 3A, address/command bus 58A isreceived by memory chip 54A. Carrier 56A has an address/command busoff-carrier connector 95 to bring signals on address command bus 58Aonto carrier 56A. An address/command bus off-carrier connector 95 isused any time an address/command bus 58 is connected to a portion of anaddress/command bus link off a carrier 56.

If a particular address/command word (see drawing of address/commandword 120 in FIGS. 5A and 5B and description later) driven by memorycontroller 52 on address/command bus 58A is not directed to (i.e. is notfor addresses on) memory chip 54A, memory chip 54A drives the particularaddress/command word (perhaps modified as will be explained later) onaddress/command bus 58B to memory chip 54M. If the particularaddress/command word driven on address/command bus 58B is not for datamemory chip 54B, memory chip 54B will drive the particularaddress/command word on address/command bus 58C to an additional memorychip (not shown). The address/command bus is “chained”, that is, eachlink of the address/command bus couples two memory chips (or the memorycontroller and a first memory chip in a daisy chain of memory chips).For example, an address/command word, if directed to a fourth memorychip 54 in a daisy chain of memory chips 47 is “chained” along the linksof address/command bus 58; that is, driven by the memory controller 52,re-driven by the first memory chip in the daisy chain of memory chips47, re-driven by the second memory chip in the daisy chain of memorychips 47, and re-driven again by the third memory chip in the daisychain of memory chips 47. Address/command busses 58 and data busses 59can be of any number of signal conductors, in various implementations,including only a single signal conductor for address/command busses 58and data busses 59. Point to point interconnection allows for very highspeed data transmission over address/command busses 58, for example, at5 GHz (gigahertz) bus clock frequency or higher, which supports 10 Gbps(gigabits per second) using DDR (double data rate) techniques.

Similarly, data busses 59 (examples shown as 59A, 59B, 59C) alsoserially couple memory controller 52 to a memory chip 54 on carrier 56A,with further serial connection to a memory chip on carrier 56B, and soon, for as many memory chips 54 as are implemented in a particular daisychain of memory chips 47. In an embodiment, data bus 59 comprises awrite portion which carries, in a daisy chained manner, data to bewritten to a memory chip 54 and a read portion which carries, in a daisychained manner, data back to memory controller 52 from a memory chip 54in the daisy chain of memory chips 47.

Carrier 56A has a data bus off-carrier connector 96 to bring signals ondata bus 59A onto carrier 56A. A data bus off-carrier connector 96 isused any time a data bus 59 is connected to a portion of a data bus linkoff a carrier 56.

Busses, such as address/command bus 58 and data bus 59 that seriallycouple interconnection of chips, such as memory controller 52 and memorychips 54 in a daisy chain of memory chips 47 are “chained” busses, eachpoint to point interconnection linking two chips being a “chained buslink”.

FIG. 3B shows an embodiment of memory system 270 comprising memorycontroller 52 and memory 210. In memory 210 of FIG. 3B, each daisy chainof memory chips 47 (daisy chain of memory chips 47A and 47B are shown,each in a dotted box) is contained on a carrier 56 (shown as carrier 56Aand 56B). A first daisy chain of memory chips 47 consists of memorychips 54A₁ to 54A₁₀. All memory chips 54 in the first daisy chain ofmemory chips 47 are attached on carrier 56A. A second daisy chain ofmemory chips 47 consists of memory chips 54B₁ to 54B₁₀. All memory chips54 in the second daisy chain of memory chips 47 are attached on carrier56B. In contrast, each daisy chain of memory chips 47 in FIG. 3Aincluded memory chips 54 attached to different carriers 56 (except, ofcourse, in a degenerate case in which a daisy chain of memory chipsconsists of but a single memory chip 54).

Address/command bus off-carrier connector 95 and data bus off-carrierconnector 96 are shown referenced at carrier 56A in FIG. 3B. Forsimplicity, address/command bus off-carrier connectors and data busconnectors 96 are typically not referenced, for example, at carrier 56Bof FIG. 3B.

In FIG. 3B, memory controller 52 drives address/command words 120(illustrated in FIGS. 5A and 5B and described in reference thereof) onaddress/command bus 58A₁ (or on address/command bus 58B₁). Anaddress/command word 120 from address/command bus 58A₁ is re-driven bymemory chip 54A₁ on address/command bus 58A₂ if the address/command word120 is not directed to memory chip 54A₁, and, as required, theaddress/command word 120 is driven down the chain of address/commandbusses 59 to memory chip 54A₁₀, as shown. A similar process occurs foraddress/command words 120 driven on address/command bus 58B₁, which isre-driven on address/command bus 58B₂ if the address/command word 120 isnot directed to memory chip 54B₁. Data words 130 (FIG. 7B, 7C) anddescription thereof are similarly chained on data busses 59A and 59B asneeded through the respective daisy chains of memory chips 47A and 47B.

It will be understood that more than a single daisy chain of memorychips 47 can be contained on a single carrier 56.

FIG. 3A and FIG. 3B, for simplicity, do not show clocking signals sentby memory controller 52. A bus clock is required to clockaddress/command busses 58 and data busses 59. A bus clock will be shownand described later. In addition, arrays on memory chips 54 requiretiming signals. Embodiments having timing signals transmitted, as wellas embodiments having memory chips 54 having self timed arrays aredescribed later.

FIG. 4 shows additional details of memory chip 54 according toembodiments of the invention. Referring to FIG. 4, memory chip 54comprises an address/command block 80, a data logic 100, and an array55. Array 55 is configured to be able to store data, memory chip 54being able to read data from and write data to, array 55.

Address/command block 80 of an instant memory chip 54 receivesaddress/command words on an address/command bus 58, denoted in FIG. 4 as58X which was driven by memory controller 52 (FIG. 3) or a memory chip54 that is upstream in the daisy chain of memory chips 47 (an “upstream”memory chip being positioned closer to memory controller 52).Address/command block 80 is configured to receive a currentaddress/command word on address/command bus 58X, and is furtherconfigured to check if the address/command word is for a read or a writein array 55 of the instant memory chip 54. If so, address/command block80 and data logic 100 are configured to perform the read/write from/toarray 55 of the instant memory chip 54. On write commands, memory chip54 is configured to send read data back on data bus 59X. If theaddress/command word received on address/command bus 58X is not for theinstant memory chip 54, address/command block 80 transmits, perhaps withcontrol modification, the address/command word to another memory chip 54on address/command bus 58Y.

It will be noted in FIG. 4 that, as data bus 59X and 59Y, inembodiments, carry data in two directions. A first direction carriesdata sent from memory controller 52 that will be written in a memorychip 54. Data read from a memory chip is carried in a second directionback to memory controller 52. In an embodiment, data bus 59X and databus 59Y are bidirectional busses. In a bidirectional bus embodiment, aparticular chip (memory controller 52 or a memory chip 54) coupled to aparticular link in the data bus 59 chain must know when the particularchip can drive the link. Control of bidirectional busses is well knownand will not be described further here.

Another embodiment of the invention comprises a data bus 59 having anoutgoing (i.e., away from memory controller 52) portion, indicated inthe shown expansions of data bus 59 as data bus 59X_(A), and data bus59Y_(A). Data bus 59, in the embodiment, further comprises an ingoing(i.e., towards memory controller 52) portion, noted as data bus 59X_(B)and 59Y_(B). In other words, subscript “A” in FIG. 4 refers to portionsof data bus 59 that carry data words 130 away from memory controller 52;subscript B refers to portions of data bus 59 that carry data words 130toward memory controller 52.

In an embodiment, data bus 59 comprises “N” bits, apportioned “M” bitsto an outgoing portion and “N-M” bits apportioned to an ingoing portion.The apportionment, in an embodiment, is fixed. For example, data bus 59has 18 bits, with nine bits in the outgoing portion (such as data bus59X_(A)) and nine bits in the ingoing portion (such as data bus59X_(B)). Alternative embodiments allow programmable apportionment ofthe bits in data bus 59 between an outgoing portion (write portion) andan incoming portion (read portion).

PLL 61 (FIG. 4) receives a bus clock 60 (60X) and re-drives bus clock 60(60Y) to a subsequent chip in the daisy chain of memory chips 47. PLL 61uses bus clock 60 to provide timing for signal transmission onaddress/command bus 58 (58X and 58Y in FIG. 4) and data bus 59 (59X and59Y in FIG. 4). PLL 61 is a Phase Locked Loop circuit in an embodimentwhere the bus clock 60 frequency needs to be multiplied to a suitablefrequency for data transmission on address/command bus 58 and data bus59. For example, if data is transmitted at 10 GB/second, double datarate, a 5 GHZ clock is needed by address/command block 80 and data logic100. If a bus clock 60 frequency is 1 GHZ, PLL 61 multiplies thefrequency on bus clock 60 by five. Alternatively, in an embodiment wherea frequency on bus clock 60 is the same frequency as data is transmittedon address/command bus 58 and data bus 59, PLL 61 is a simple buffercircuit.

Timing block 63 (FIG. 4) is required on memory chips 54 that do notimplement self timing of array 55. Memory chips 54 that do implementself timing of array 55 will be described in detail later. Timing block63 receives timing signals 62 (62X) that originate from memory control52 and the timing signals 62 are chained on point to pointinterconnections through the daisy chain of memory chips 47. Forexample, timing signals 62 provide timings for precharging bit lines(not shown) in array 55, driving word lines (not shown) in array 55. Ingeneral, timings depend on particular implementation of array 55. Timingblock 63 re-drives timing signals 62 (62Y) to the next memory chip 54 inthe daisy chain of memory chips 47.

In many scientific applications, writes are as common as reads, andadvantageously there is an equal apportionment of bandwidth for writedata and read data. However, in many commercial applications, readsgreatly outnumber writes and a more advantageous apportionment is toprovide more bandwidth for read data than for write data. For example,if data bus 59 has 18 bits, an appropriate apportionment may be twelvebits allocated to the incoming (read) portion (data bus 59X_(B)) and sixbits allocated to the outgoing (write) portion (data bus 59X_(A)).Furthermore, because processor 200 is often stalled while waiting fordata, reads are usually prioritized over writes; a larger number of bitsin data bus 59 should be apportioned to the incoming portion of data bus59.

In a further embodiment, data logic 100 implements a programmableapportionment of the “N” bits so that a memory chip 54 and a memorycontroller 52 can be tuned (e.g., by conventional scan data or pinconnection) for a computer system 250 that runs a preponderance ofscientific applications (with read data bandwidth similar to write databandwidth) or a preponderance of commercial applications (with read databandwidth being greater than write data bandwidth). An example showinghow data bus 59 is programmably apportioned is given later withreference to FIGS. 7E and 7F.

FIG. 5A shows an exemplary address/command word 120 transmitted onaddress/command bus 58. Address/command word 120, in the embodimentshown in FIG. 5A comprises a chip ID 121, a command 122, a packet number123, and an address 124. Memory controller 52 knows which memory chip 54in a particular daisy chain of memory chips 47 a particular piece ofdata is to be written to or read from. Chip ID 121 identifies whichmemory chip 54 in a particular daisy chain of memory chips 47 theaddress command word 120 is intended for, and the chip ID 121 is writtenby memory controller 52 when the address/command word 120 is sent frommemory controller 52. Chip ID 121 can be a binary number. For example,if there are eight memory chips 54 in a particular daisy chain of memorychips 47, chip ID 121 has a value “000” if the address/command word 120is intended for the first memory chip 54; a value “001” if theaddress/command word 120 is intended for the second memory chip 54, andso on. Each particular memory chip 54, upon receipt of a particularaddress/command word 120 compares the chip ID 121 in the address/commandword 120 against a memory chip ID known to the particular memory chip54. The memory chip ID, in embodiments, is scanned in throughconventional scan techniques at system bring up into each memory chip54, or can be programmed by way of having pins on each memory chip 54being coupled to suitable voltages. For example, the first memory chip54 in a daisy chain of eight memory chips 54 has three memory chip IDpins, all connected to a logical “0”. The last memory chip 54 in thedaisy chain of eight memory chips 54 has its three memory chip ID pinsall connected to a logical “1”.

Command 122 is typically a simple “read” or “write” indicator, forexample, a logical “0” for read and a logical “1” for a write.

Packet ID 123 is a packet identifier field in an address/command word120. Packet ID 123 contains a value assigned by memory controller 52 toassociate a particular address/command word 120 with a particular dataword (to be described later). The size (i.e., number of bits) of packetID 123 depends, in a particular design, upon how many outstandingread/write requests (commands) memory controller 52 is designed tosupport on a particular daisy chain of memory chips 47. For example, ifmemory controller 52 supports sixteen outstanding requests in a daisychain of memory chips 47, four bits are required. Memory chips 54 maysupport a larger number of bits (e.g., have an eight bit field availablefor packet ID 123 data); if so, memory chips 54 will be programmedduring bring up to realize that fewer than eight bits are used forpacket ID 123 data. Alternatively, memory controller 52 simply transmits(using the example above) four bits of packet ID 123 data, with anadditional four bits of padding (e.g., zeros).

Address 124 is the location in the array of memory chip 54 whererequested data is to be written to or read from. As with packet ID 123,length of address 124 can be programmable or address bits can be paddedby memory controller 52.

CRC 125 is a cyclic redundancy code field used in some embodiments.Implementations requiring CRC for reliability or other reasons includeCRC 125. If CRC is not required, CRC 125 is not implemented, or is notfilled with valid data or checked for valid data if implemented.

FIG. 5B shows an alternative embodiment of address/command word 120. Inalternative embodiments of the invention, data to be written into anarray 55 of a memory chip 54 is transmitted with command word 120. Suchalternative embodiments eliminate the need for “outgoing” data bus 59portions, such as 59X_(A) and 59Y_(A), shown in FIG. 4. However,transmitting data to be written into an array 55 significantly increasesbandwidth requirements on address/command bus 58, and in suchembodiments, address/command bus 58 is typically made wider thanembodiments in which data to be written into an array 55 is transmittedon a separate “outgoing” data bus such as 59X_(A) and 59Y_(A), shown inFIG. 4. Address/command word 120 shown in FIG. 5B comprises chip ID 121,command 122, packet ID 123, address 124, and an optional CRC 125 asexplained in reference to CRC 125 in address/command word 120 in FIG.5A, and further comprises write data 126. Unless specified, forsimplicity, an address/command word 120 as shown in FIG. 5A will behereinafter assumed.

It will be further noted that, in FIG. 5C, no packet 123 is implemented.Whereas in many embodiments of the invention to be described later,packet 123 is required to identify which data word 130 is associatedwith a particular address/command word 120, in other embodiments, memorycontroller enforces timing constraints such that ambiguities betweendata words 130 and address/command words 130 do not exist.

FIGS. 6A-6D illustrate address/command 80 in more detail. FIGS. 6A, 6B,and 6C show an embodiment in which address/command bus 58 is a singlebit wide. FIG. 6D shows that in an alternative embodiment,address/command bus 58 contains a plurality of bits, with appropriatelogic in address/command 80 in an instant chip configured to recognizewhen an address/command word 120 is directed to the instant chip.

Referring now to FIG. 6A, address/command block 80 is configured toreceive address/command words 120 over address/command bus 58X. As shownin FIGS. 5A and 5B, the first bits coming in on an address/command word120 are the chip ID bits 121. Advantageously in a one-bit-wideaddress/command bus 58 embodiment where the number of memory chips in adaisy chain is not long (e.g., over about eight memory chips) a“one-hot” implementation of chip ID 121 is used. FIG. 6B shows memorycontroller 52 and a daisy chain of memory chips 47 having four memorychips 54 (54 ₁-54 ₄). Address/command busses 58 ₁-58 ₄ are used totransmit address/command words 120, as shown in FIG. 6B. Data busses 59₁-59 ₄ are used to send/receive data to/from memory chips 54 ₁-54 ₄. Inthe embodiment shown, a chip ID 121=“0001” (where the “1” is the firstbit transmitted) sent by memory controller 52 will reference the firstmemory chip 54 (i.e., memory chip 54 ₁) in the daisy chain of memorychips. A chip ID 121 value of “0010” sent by memory controller 52references the second memory chip 54 in the daisy chain, and so on. Insuch an embodiment, each particular memory chip 54 will handle theincoming address/command word 120 if the leading bit in the incomingaddress/command word 120 is “1”. Each memory chip 54 shifts the chip ID121 by one bit.

As shown in FIG. 6A, in address/command block 80, block 83 checks if theleading bit in an address/command word 120 is “1”. If so, theaddress/command word 120 is written into an address/command queue 81.Address/command queue 81 contains one or more address/command buffers 82(address/command buffers 82A-82C shown).

If the leading bit of the incoming address/command word 120 is not “1”,the address/command word 120 is simply routed through a shift/pad chipID 84 onto address/command bus 58Y, with the bits in chip ID 121 shiftedby shift/pad chip ID 84 by one bit to the right, with padding added onthe left. For example if memory controller 52 sends an address commandword 120 on address command bus 58 ₁ (FIG. 6B) having a chip ID 121value of “0010”, the chip ID 121 driven by memory chip 54 ₁ ontoaddress/command bus 58 ₂ will have a value of “0001”, and memory chip 58₂ will handle the request.

FIG. 6C shows how the contents of chip ID 121 are shifted to the rightin an address/command word 120 intended for memory chip 54 ₄. Thisshifting technique allows an instant memory chip 54 to immediatelyrecognize whether a particular address/command word 120 received on aone bit wide address/command bus 58 is directed to the instant memorychip 54 or needs to be passed to the next memory chip 54 in the daisychain of memory chips 47. If such shifting were not done, the entirecontents of the chip ID 121 field would have to be received by aninstant memory chip 54 before the instant memory chip 54 would know ifthe incoming address/command word 120 is directed to the instant memorychip 54 or must be re-driven to the next memory chip 54, causing delaysand requiring additional buffering of address/command word 120.

FIG. 6D shows an embodiment of address/command 80 similar to that shownin FIG. 6A except that address/command word is transmitted over anaddress/command word 58 having more than a single bit. Chip ID compare87 compares chip ID 121 received on address/command bus 58X against thememory chip ID for the particular memory chip 54. The memory chip ID, inembodiments is scanned into a register (not shown), or programmed usingI/O pins (not shown) on the particular dread chip 54. If a chip ID 121matches the memory chip ID, the address/command word 120 is placed inaddress/command queue 81; if not, the address/command word is re-drivenon address/command bus 58Y.

Embodiments of address/command 80 that shift chip ID that also includeCRC 125 must ensure that the value in CRC 125 remains valid. Forexample, in an embodiment, CRC 125 is regenerated in shift/pad DRAM ID84. In an alternative embodiment, a value in CRC 125 does not includechip ID 121; that is, CRC generation does not consider chip ID 121.

Array fetch/store logic 85 (FIG. 6A and FIG. 6D) processes commands onmemory chip 54 in a conventional manner, using addresses 124 fromaddress/command words 120 in address/command buffers 82 inaddress/command queue 81. Addresses and other control signals (such asto control bit line precharge, word line enable, etc, depending onparticular design details of a particular array 55) are sent to array 55on signals 88. Signals 93 communicate information to data logic 100 asto whether data is being written to or read from array 55, along withvalues of each particular packet ID 123 so that data read/written fromarray 55 is always associated with the particular packet ID 123.

FIG. 7A shows details of an exemplary data logic 100. Data control 140is in communication with address/command block 80 via signals 93, andfurther with array 55 via signals 94 (see FIG. 4). Data control 140manages reads and writes to array 55. For example, if a currentlyselected address/command word 120 in address/command queue 81 requestsdata from array 55, data control 140 will cause a read operation tooccur to array 55. When the data read is received from array 55, thedata is routed on signals 94 to read queue 141. The associated packetvalue from the packet ID 123 of the selected address/command word 120 isappended to the data received from array 55 in an add packet ID 142,forming a data word 130, as shown in FIG. 7B. A data word 130 comprisesa data portion 131 and a packet ID portion 132 as shown in FIG. 7B.

A data word 130 containing data 131 that has been read from an array 55in a memory chip 54 is called a read data word. A data word 130containing data 131 that is to be written into an array 55 in a memorychip 54 is called a write data word.

As with address/command word 120, in embodiments, a CRC 135 portion isincluded in data word 130 and contains cyclic redundancy codeinformation to enhance reliability of data transmission. In embodimentsimplementing CRC 135, generation of a CRC value for a particular dataword 130 is performed in add packet ID 142.

If a write operation is indicated by the currently selectedaddress/command word 120, write queue 145 is checked for a packet ID 132in write queue 145 that matches the packet ID 123 of the address/commandword 120. If found, data 131 from a data word 130 having the matchingpacket ID 132 is written to the address 124 of the currently selectedaddress/command 120 word. If the packet ID 123 is not matched in writequeue 145, data control 140 waits for a predetermined time interval,during which data control 140 can service other address/command words120. If, after the predetermined time interval, a packet ID matchbetween a particular packet ID 120 and any packet ID in write queue 145is not successful, an error may be reported back to memory controller52. For example, packet ID 132, in an embodiment, contains one more bitthan packet ID 123. The extra bit is set to “1”, and the packet ID 123value is sent back to memory control 52 in the remainder of packet ID132, on data bus 59X. Memory controller 52, upon receipt of a data word130 containing a “1” in the extra bit of the packet ID 132 knows thatthe data sent for that packet was not successfully written into a memorychip 54. As before, in an embodiment as illustrated in FIG. 4A, data bus59 contains one or more incoming signals, e.g., 59X_(A), and one or moreoutgoing signals, e.g., 59X_(B).

Data control 140 also manages write queue 146 and read queue 147 shownin FIG. 7A. Write queue 146 contains write data words 130 (FIGS. 7B and7C) having data destined to be written to memory chips 54 further downthe daisy chain of memory chips 47. Read queue 147 contains read datawords 130 having data that has been read from memory chips 54 furtherdown the daisy chain of memory chips 47.

As data words 130 arrive on data bus 59X, packet ID 132 values arechecked against packet ID 123 values. If a match occurs, the data word130 is routed to write queue 145; if not matched, the data word 130 isrouted to write queue 146 for further transfer to memory chips 54further down the daisy chain when data bus 59Y is available. A data 131from a particular data word 130 is written into array 55 when writequeue 145 is able to do so (that is, when array 55 is not beingotherwise accessed and when whatever priority arbitration mechanism in aparticular implementation selects the particular data word from writequeue 145.

Similarly, data words 130 are received from data bus 59Y and are routedto read queue 147 and further routed towards memory control 52 when databus 59X is available for transmitting in the direction of memorycontroller 52.

Some embodiments of the invention include a chip ID portion of data wordwhich simplifies the task performed by data control 140 on an instantmemory chip in determining whether a particular data packet is for theinstant memory chip 54. A data word 130 as shown in FIG. 7C comprisesdata 131, packet ID 132, and chip ID 133. In such an embodiment, datacontrol 140 of an instant memory chip 54 knows a particular data word130 is directed to that instant memory chip, and does not have to matchpacket IDs 123 to packet IDs 132 to know whether a particular data wordis directed to the instant memory chip 54. Chip ID 133 will have thesame value, for a given chip in a daisy chain of memory chips 47, aschip ID 123 in an address/command word 120.

In embodiments, data word 130 in FIG. 7C includes CRC 135 which containscyclic redundancy code information to enhance data transmissionreliability. Chip ID 133 is written by memory controller 52 for writedata words 130 having data 131 to be written to a memory chip 54. ChipID 133 need not be written into by a memory chip 54 when the memory chip54 writes into a particular read data word 130 since the read data word130 written into by a memory chip 54 is destined for memory controller52, not any of the other memory chips 54 in the daisy chain. As above, avalue in CRC 135 is generated by add packet ID 142.

FIG. 7D shows a data logic 100 suitable for embodiments in which data tobe written into an array 55 of a memory chip 54 is sent as part of anaddress/command word 120 as described earlier, that is, write data 126,shown in FIG. 5B. Data control 140 in FIG. 7D is similar to data control140 in FIG. 7A, but receives write data 126 on signals 93 (FIG. 5B) tobe written into array 55 (see FIG. 4) on signals 94. Data control 140passes write data 126 and packet ID 123 to write queue 145. A particularwrite data 126 is written into array 55 when write queue 145 is able todo so (that is, when array 55 is not being otherwise accessed and whenwhatever priority arbitration mechanism in a particular implementationselects the particular write data 126 from write queue 145.

Read queue 141, add packet ID 142, and read queue 147 are as explainedin reference to FIG. 7A.

Programmable apportionment of data bus 59 was introduced earlier, andwill be described in detail now with reference to FIGS. 7E and 7F. Asstated earlier, many scientific applications require approximately thesame bandwidth for writes as for reads. In such applications, data bus59X should have approximately the same number of signal conductorscarrying write data words 130 outward for writes to a memory chip 59 assignal conductors carrying read data words 130 inward for reads from amemory chip 59. Other applications, such as commercial applications,need considerably more bandwidth for reads than for writes.

FIG. 7E shows a block diagram of a data logic 100 that provides aprogrammable apportionment of data bus 59 between an outgoing portionand an incoming portion. A read/write apportion register 148 contains adesired apportionment. The desired apportionment can be scanned intoread/write apportion register 148 at bring up of computer system 250.Alternatively, an application running on processor 200 can requestmemory controller 52 to write read/write apportion registers 148 in eachdaisy chain of memory chips 47 connected to the memory controller 52.Memory controller 52 and memory chips 54 must, at any given time, be inagreement as to how data bus 59 is apportioned between an outgoingportion and an ingoing portion. In an alternative embodiment, pins oneach memory chip, and memory controller 52 are simply connected toappropriate voltages to provide a fixed apportionment. In such analternative embodiment, the logical value supplied by the connection ofpins to appropriate voltage values performs the same role as read/writeapportion register 148.

Alternatively, apportionment of data bus 59 is done by transmission of adesired apportionment from processor 200 to memory controller 52. Memorycontroller 52 subsequently transmits the desired apportionment to memorychips 54 (e.g., in an address/command word 120 having a suitable command122 and an apportionment value in a portion of address 124).

In another alternative embodiment, memory controller 52 has one or moreI/O pins connected to sources of logical values that, when powered,program the apportionment at memory controller 52. Memory controller 52must forward the apportionment to memory chips 54, such as the memorychips 54 in a daisy chain of memory chips 47. Memory controller 52creates an address/command word 120 having a command value in command122 that memory chips 54 are configured to recognize an apportioncommand. Memory controller 52 fills in a value in a portion of address124 that memory chips 54 use to make the apportionment at the memorychip 54.

A read/write I/O block 500 contains circuits that are receivers only,drivers only, or common I/Os. A common I/O is a circuit that can eitherbe a driver or a receiver based on a value of a control signal.Read/write I/O 500 is coupled to read/write apportion register 148 viasignals 149. Read/write I/O 500 provides for receiving write data words130 on data bus 59X, where, as before, data bus 59X is on a proximalside, logically, to memory controller 52. Read/write I/O 500 alsoprovides for driving read signals back in the direction of memorycontroller 52 on data bus 59X. A read/write I/O 550 block provides afunction similar to read/write I/O block 500, but for data bus 59Y,where, as before, data bus 59Y is on a distal side, logically, frommemory controller 52. Read/write I/O 550 is controlled by read/writeapportion register 148 by signals 150. Other referenced numbered blocksin FIG. 7E are as described for like numbered blocks described earlier.

FIG. 7F shows in more detail how read/write apportion register 148controls apportionment of data bus 59.

Data bus 59X is shown, for exemplary purposes, to have six signalconductors (59X-1 to 59X-6). I/O circuits 501-506 drive and/or receivesignal conductors 59X-1 to 59X-6 as shown in FIG. 7F and describedherein.

Data bus 59X, in a first apportionment, has two signal conductors (59X-1and 59X-2) apportioned to write data, circled and referenced as 5OX_(A)1. I/O circuits 501 and 502 are controlled by signals 149 to bereceivers. In the example, I/O circuits 501 and 502 are never used asdrivers, and could alternatively be designed as receivers only. In thefirst apportionment, four signal conductors (59X-3, 59X-4, 59X-5, 59X-6)are apportioned to read data, with these four signal conductors circledand referenced 59X_(B) 1. Read/write apportion register 148 controlssignals 149 to control I/O circuits 503, 504, 505, and 506 to be driverswhich will drive data from reads on signals 59X-3, 59X-4, 59X-5, and59X-6 back in the direction of memory controller 52. In the firstapportionment, data bus 59 uses twice as many signal conductors to driveread data back towards memory controller 52 as are used to drive writedata outwards to a memory chip 54.

In the first apportionment, data received by I/O circuits 501 and 502are sent on signals W1 and W2 to write queue 145 and are written into adata word in write queue 145 under control of read/write apportionregister 148 via signals 510. Signals 510 instruct write queue 145 toaccept W1 and W2 and then shift the data word by two bit positions inorder to receive two new signals on W1 and W2. This process is repeateduntil a data word 130 is filled. A similar process is used to move datafrom signals 59X-1 and 59X-2 into write queue 146 when a write data word130 is not directed to the particular memory chip 54 and therefore needsto be re-driven to another memory chip 54.

In some embodiments, a write data word 130 is shifted into a temporarybuffer (not shown) and the data word 130 is transferred to write queue145 or write queue 146 (FIG. 7A) once it is determined if the write dataword 130 is to be written to the present memory chip 54 or needs to bere-driven.

In the first apportionment, four bits at a time from read queue 147(R1-R4) are sent to I/O circuits 503-506, respectively. Read/writeapportion register 148 controls I/O circuits 503-506 to drive signalconductors 59X-3, 59X-4, 59X-5, and 59X-6 with the values on R1-R4.Read/write apportion register, via signals 511, controls read queue 147to shift by four bits each cycle in order to present the next four bitsof a data word 130 stored in read queue 147.

In a second apportionment, data bus 59 is apportioned equally betweenoutgoing data and incoming data. That is, signals 59X-1, 59X-2, and59X-3 are used for write data, as shown grouped as 59X_(A) 2. Incomingread data is transmitted back in the direction of memory controller 52on signal conductors 59X-4, 59X-5, and 59X-6, as shown grouped as59X_(B) 2. Read/write apportion register 148 controls I/O circuits 501,502, 503 to be receivers; I/O circuits 504, 505, and 506 to be drivers.In the example, I/O circuits 504, 505, and 506 are always used asdrivers, and a receiver portion of I/O circuits 504, 505, and 506 is notrequired. Test requirements, in some applications, may require that allI/O circuits (501-506) be common I/O circuits, whether functionallyrequired or not.

In the second apportionment, I/O circuits 501, 502, and 503 send signalsW1, W2 and W3 to write queue 145. Read/write apportion register 148, viasignals 510, causes the selected data word register in write queue 145to shift three bits at a time. Similarly, read/write apportion register148, via signals 511, causes the selected register in read queue 147 toshift three bits at a time, in order to send three new bits of data word130 via R1, R2, R3 to I/O circuits 504, 505, and 506 to be driven onsignal conductors 59X-4, 59X-5, and 59X-6. Data words from add packet ID142 are treated in a similar manner to data words from read queue 147.

Circuitry and control of read/write I/O 550 is similar to circuitry andcontrol of read/write I/O 500. Read/write apportion register 148, viasignals 150 controls which signals on data bus 59Y are used for writedata words 130 and which signals on data bus 59Y are used for read datawords 130. Memory controller 52 also contains similar circuitry andcontrol to apportion data bus 59 connected to memory controller 52between an outgoing portion used for write data words 130 and anincoming portion used for read data words 130.

Programmable apportionment of a data bus 59 between a first portiondedicated to write data and a second portion dedicated to read data canalso be expressed as a method embodiment.

Referring now to FIG. 19, method 900 is shown. Method 900 begins at step902. In step 904, data bus 59 is apportioned into a first portiondedicated to write data and a second portion dedicated to read data. Instep 906 the data bus is used according to the current apportionment;that is, write data is transmitted to a memory chip 54 using the firstportion of the data bus, and read data is transmitted from a memory chip54 to memory controller 52 using the second portion of the data bus. Inan embodiment having a fixed programmable apportionment (e.g.,programmed by connection of I/O pins to appropriate sources of logicalvalues) the method ends and the apportionment is not changed.

In an embodiment where apportionment is dynamically programmable, suchas upon a request for a change of apportionment by processor 200,control passes to step 908. If no change in apportionment is requested,control passes back to step 906. If a change in apportionment isrequested, control passes to step 910, which suspends transmission datawords 130. In an embodiment, suspending transmission of data words 130includes sending one or more address command words 120 having a commandthat alerts memory chips 54 that a reapportionment of data bus 59 ispending and to stop sending read data words 130. Memory controller 52allows sufficient time for any already transmitted write data word 130to be received by the proper memory chip 54 and any already transmittedread data word to be received by memory controller 52, then sendsanother address/command word 120 having information as to the newapportionment.

In step 912, data bus 59 is re-apportioned between the first portionused for write data and the second portion used for read data. Uponcompletion of the re-apportionment, control passes to step 914, whichresumes transmission data words 130 on data bus 59.

Embodiments of the invention implement address/command bus 58 and databus 59 interconnection of a daisy chain of memory chips 47 as shown inFIG. 8. FIG. 8 shows memory system 270 having memory controller 52driving address/command bus 58A and unidirectional data bus 59A. Databus 59A does not have two portions as depicted in FIG. 4A (59X_(A),59X_(B)), but, rather, is a single, unidirectional data bus. As shown inFIG. 8, a number of memory chips 54 (memory chips 54A-54M) are attachedon carrier 56.

Data words 130 (data words shown in FIGS. 7B and 7C) are driven ontodata bus 59A at data bus port 32 (one shown referenced in FIG. 8) bymemory controller 52 for writing into a memory chip 54. Data words 130having data 131 read from a memory chip 54 are received by memorycontroller 52 on data bus 59N. For example, memory controller 52 needsto write data into memory chip 54A and read data from memory chip 54B.An address/command word 120 is transmitted at address/command bus port31 (one shown referenced in FIG. 8) on address/command bus 58A and isrecognized by memory chip 54A as a write into its array 55 for datahaving a packet ID value matching the value in packet ID 123 of theaddress/command word 120. As described earlier, data 131 from a writedata word 130 having a packet ID 132 matching the packet ID 123 ofaddress/command word 120 is written into the array 55 of memory chip54A. The address/command word 120 for the write into memory chip 54A, inan embodiment, is not forwarded, as the succeeding memory chips 54 inthe daisy chain do not require it. In a second embodiment, theaddress/command word 120 for the write into memory chip 54A is passed tosubsequent memory chips in the daisy chain on address/command busses 58(e.g., 58B) and the address/command word 120 is returned onaddress/command bus 58N to memory controller 52, confirming to memorycontroller 52 that the address/command word was successfullytransmitted.

Continuing the example of the embodiment shown in FIG. 8, memorycontroller 52 subsequently sends an address/command word 120 requestinga read from memory chip 54B. Memory chip 54B recognizes and receives therequest and, when the requested data has been read from array 55 onmemory chip 54B, and, when the data logic 100 of chip 54B determinesthat data can be transmitted on data bus 59C, the appropriate read dataword 130 is transmitted on data bus 59C and continues through the daisychain and is received on data bus 59N by memory controller 52. Bus clock60A and timing signals 62A are likewise chained through daisy chain ofmemory chips 47A-54M, and, at the end of the daisy chain of memory chips47A-54M are shown being returned to memory controller 52 as bus clock60N and timing signals 62N. Bus clocks 60 are brought onto or drivenfrom memory 52 at bus clock port 33 (one shown referenced in FIG. 8).Timing signals 62 are driven from or received by memory controller 52 attiming signals port 34 (one shown referenced in FIG. 8).

Bus clocks 60 are carried onto or off of a carrier 56 via bus clockoff-carrier connectors 98, one of which is referenced in FIG. 8. Timingsignals 62 are carried onto or off of a carrier 56 via timing signalsoff-carrier connectors 99, one of which is referenced in FIG. 8.

Each memory chip 54 in a daisy chain of memory chips 47 can beprocessing requests for reads and/or writes in parallel. For example,memory controller 52 may issue a read request (using an appropriateaddress/command word 120 transmitted on address/command bus 58A) formemory chip 54A, with memory controller 52 sending subsequent additionalread (or write) requests to other memory chips 54 in the daisy chain ofmemory chips 47 before memory chip 54A has fetched the requested datafrom array 55 in memory chip 54A. In addition, multiple read (or write)commands can be queued up in each particular memory chip 54 as explainedin the discussion on address/command queue 81 in FIG. 6A, and the readand write queues described in FIG. 7A. Some memory chips 54 may comprisemultiple banks of memory that can be accessed independently from eachother. In such memory chips 54, multiple reads and/or writes can be inprogress at the same time on the same memory chip 54.

Such parallel handling of requests can occur when transmission ofaddress/command words 120 and transmission of data words 130 are fastrelative to read and write times in arrays 55 on memory chips 54. Forexample, if reads and writes to arrays 55 take eight times longer than atransmission of an address/command word 120 or a data word 130, it ispossible to have each memory chip 54 in an eight memory chip 54 daisychain processing reads and writes at once. Currently, read and writetimes of DRAM memory chips is about 40 ns (nanoseconds). A twenty bitaddress/command word 120 transmitted at 10 Gb/s takes two nanoseconds totransmit, assuming a single bit wide address/command bus 58.

FIG. 9A shows a further embodiment of the invention. FIG. 9A showsmemory system 270 having memory controller 52 coupled to memory chips 54on carrier 56, as was shown in FIG. 8. The embodiment shown in FIG. 9Adoes not need timing signals 62 (see FIG. 8) since arrays 55 on memorychips 54 are self timed in the embodiment depicted in FIG. 9A. Bus clock60 controls the speed at which data is received and transmitted onaddress/command busses 58 and data busses 59. In prior figures anddescriptions, a conventional clocking scheme was assumed (e.g., usingtiming signals 62 and timing block 63 as shown in FIG. 4). It will beunderstood that, where self timed memory chips are used, timing signals62 are not required to be transmitted by memory controller 52 andchained through a daisy chain of memory chips 47. In some prior figures,for simplicity, bus clock 60 was not shown (e.g., FIGS. 3A, 3B, and 6B).

FIG. 9B is a block diagram of an exemplary memory chip 54 that has aself timed array 55. As explained above, bus clock 60 provides for a busfrequency at which data is received and transmitted on address/commandbusses 58 and data busses 59. Clock bus 60 may provide a clockfrequency, e.g., 5 GHz (gigahertz) at which data is sent or received onaddress/command busses 58 and data busses 59. Often, in high speedlinks, a 5 GHz bus clock, using double data rate techniques, wouldresult in a 10 gigabit per second data transfer on each signal conductorin a bus. Alternatively as shown in FIG. 9B a lower frequency clock canbe sent on bus clock 60 and transformed on each memory chip 54 via aPhase Lock Loop (PLL) 61 to the appropriate frequency of address/commandbusses 58 and data busses 59. A self time block 300 is used to read datafrom and write data to array 55. Timing signals 62 and timing block 63are not implemented in a self timed embodiment of memory chip 54.

Array 55 comprises bit lines (not shown) that need to be precharged bybit line precharge circuits (not shown) and discharged by memory cells(not shown). Precharge and discharge times tend to differ between afirst memory chip 54 and a second memory chip 54 due to processvariations when the first memory chip 54 and the second memory chip 54were produced. Furthermore, array 55 access times are also dependent ontemperature of array and voltage applied to array 55.

In the daisy chain of memory chips 47 shown in FIG. 8, all memory chipsare clocked at the same frequency, that is, by timing signals 62. Thatfrequency typically accommodates a slowest memory chip 54 at designerspecified worst case voltage and temperature conditions.

The embodiment illustrated in FIGS. 9A and 9B show a particular memorychip 54 to have array 55 on that particular memory chip 54 to be readfrom or written to at a rate determined by actual access times whichinclude actual precharge and discharge rates of the array 55 on theparticular memory chip 54.

Self time block 300 is coupled to array 55, address/command 80 and datalogic 100 on self time signals 301, 302, and 303. Self time block 300,as described in detail below, will cause arrays 55 that can be accessedfaster to be accessed faster, and arrays 55 that are relatively slowerin performance to be accessed more slowly.

For example, if a particular daisy chain of memory chips 47 has eightmemory chips 54, and a first memory chip in the particular daisy chainof memory chips 47 takes 60 ns (nanoseconds) to access, while theremaining seven of the memory chips 54 in the particular daisy chain ofmemory chips 47 require only 30 ns to access, average access rates ofthe particular daisy chain of memory chips 47 that feature a self timeblock 300 (that is, are self-timed memory chips 54) are almost twice asfast as a daisy chain of memory chips 47 that are not self timed memorychips 54. In other words, seven of the eight memory chips 54 access in30 ns; only one accesses at 60 ns.

Self time block 300 contains a ring oscillator for DRAM performancemonitoring. U.S. Pat. No. 6,774,734 teaches a ring oscillator having afrequency determined by a dynamic memory performance.

FIG. 10 shows self time block 300 further comprising a DRAM ringoscillator 305. DRAM ring oscillator 305, in embodiments, is DRAM ringoscillator 700 seen in FIG. 7 of U.S. Pat. No. 6,774,734, or multiplexedDRAM ring oscillator 800 seen in FIG. 8 of U.S. Pat. No. 6,774,734. DRAMring oscillator 305 is implemented to be of similar design to array 55so that access times will track.

Signal 306 in FIG. 10 couples an output of DRAM ring oscillator 305 toArray timing control 350, which provides self time signals, 302, and 303introduced in FIG. 9B.

Embodiments of Array timing control 350 are seen in more detail in FIGS.11A and 11B. FIG. 11A shows signal 306 coupled to scaler 352. Scaler 352adjusts the frequency of the signal from DRAM ring oscillator 305. Suchadjustment is typically needed to provide appropriate granularity ofcontrol self time signals, 302, 303. For example, if DRAM ringoscillator 305 produces a frequency similar to a maximum frequency ofaccess of array 55, scaler 352 will need to produce a higher frequencyoutput in order to provide appropriate sub timings needed in typicaldesigns of array 55.

A PLL embodiment of scaler 352 provides such a higher frequency. FIG.11A shows scaler 352 providing a clock to register 307. Register 307contains a single “1” with remaining bits “0”. Register 307 wraps afinal bit position to a first bit position, so that the “1” willconstantly circulate at a frequency determined by DRAM ring oscillator305 (as scaled by scaler 352). Address/command block 80 and data logic100 are in communication with register 307 and can enable or disableshifting. For example, in absence of a request, the “1” will be held ina first bit position of register 307. when a request is started,address/command block 80 and/or data logic 100 enable shifting ofregister 307, and the “1” will then advance through register 307.

The “1” will remain in each bit position of register 307 for a timeproportional to the frequency of DRAM ring oscillator 305. Signal 303,taken from a first bit position in register 307, latches an address(from a currently selected address/command word 120) into a word selectcircuit (not shown) in array 55 (FIG. 9B). A self time signal 301A inself time signal 301 activates a bit line precharge (not shown) in array55 (FIG. 9B). A second self time signal 301B in self time signal 301activates a word line activate which causes selected bit lines (notshown) to discharge in array 55 (FIG. 9B). OR 353 produces self timesignal 301A, which remains “1” when a “1” is in a second, third, fourth,or fifth bit position in register 307. Similarly, OR 354 produces selftime signal 301B, which remains “1” when a “1” is in a seventh, eighth,ninth, or tenth position of register 307. An eleventh bit in register307 is used for self time signal 302 which latches data read from array55 into data logic 100. It will be understood that self time signals301, 302, and 303 are exemplary only to illustrate how array timings ona memory chip can be made responsive to a ring oscillator, a frequencyof the ring oscillator dynamically dependent on array timings. Actualnumber of self time signals required and required timings of suchsignals depends on details of a particular design of array 55.

A second embodiment of Array timing control 350 is shown in FIG. 11B. Inthis embodiment, timings of self time signals 301, 302, and 303 areagain dependent on a frequency of DRAM ring oscillator 305. Thefrequency of DRAM ring oscillator 305, as above, tracks a rate at whicharray 55 can be accessed.

In the embodiment of FIG. 11B, it is assumed that the bus clockfrequency is significantly higher than the frequency of DRAM ringoscillator 305, the difference in frequencies being sufficient toprovide appropriate granularity of control of self time signals 301,302, and 303. When signal 306 (output of DRAM ring oscillator 305) is“high” (half the period of the signal on signal 306), bus clock 60Yincrements counter 360. When signal 306 falls, a current value ofcounter 360 is captured in register 362, and counter 360 is reset. Thevalue captured in register 362 is relatively larger when DRAM ringoscillator 305 is “slow”, and relatively smaller when DRAM ringoscillator 305 is “fast”. DRAM ring oscillator 305 is, as stated above,designed to track performance of array 55. A decode 364 uses bus clock60Y and the value captured in register 362 to provide timings of selftime signals 301, 302, and 303. For example, if register 362 has arelatively large value, array 55 is relatively slow, and a relativelylong precharge time and discharge time is required to access (read orwrite) to array 55. Alternatively, if register 362 contains a relativelylow value, precharge and discharge of array 55 is relatively fast, andarray 55 can be accessed at a faster rate.

Different bus clock 60Y frequencies may be used in different memorysystems. Decode 364 is advantageously programmable (e.g., byconventional scan-in of bus clock frequency information) to accommodatedifferent bus clock frequencies.

Timings of each chip 54's accesses to array 55 is asynchronous with busclock 60. Conventional synchronization techniques are used inaddress/command 80 and data logic 100 (FIG. 9B) when providing for datatransmission on address/command bus 58 and/or data bus 59.

In some embodiments of the invention, memory controller 52 transmitsaddress/command words 120 timed such that collisions do not occur ondata bus 59. In a daisy chain of self timed memory chips, memorycontroller 52 is configured to periodically request, using anappropriate command 122 in an address/command word 120, to a memory chip54 to transmit a data word 130 having an update on access timings of thearray on memory chip 54. In an embodiment, memory chip 54 is not allowedto change access times of array 55 until authorized to do so by anotheraddress/command word 120. In such an embodiment, a “new” access timevalue is held in a temporary storage (not shown), the “new” access timebeing applied to array access timings upon approval by memory controller52.

Data bus 59 typically requires more bandwidth than address/command bus58. For example, address/command bus 58, in a particular implementation,carries address/command words 120 which further comprise (see FIG. 5) afour-bit chip ID 121, a one-bit command 122, an eight-bit packet ID 123,and a sixteen-bit address 124. Each command/address word 120 isassociated with a data word 130 (see FIG. 7B, 7C) which, for example,comprises a four-bit chip ID 133, an eight-bit packet ID 132, and a64-bit data 131. In the example, command/address word 120 contains 29bits; data word 130 contains 76 bits. In an embodiment, data bus 59 issimply made proportionally wider than a corresponding address/commandbus 58. In the example, data bus 130 is made 76/29 times the width ofaddress/command bus 58. It will be understood that the bus width ratiosof address/command bus 58 to data bus 130 need not be exactly the sameas the required bandwidth. In the example, a ratio larger than 76/29would tend to reduce memory latency as data busses 59 would be lesscongested.

An alternate method of balancing bandwidth requirements onaddress/command bus 58 with bandwidth requirements on data bus 59 isshown in FIG. 12. FIG. 12 shows an exemplary memory system 270 in whicha daisy chain of memory chips 47 has an address/command bus 58 chainthat chains through all the memory chips 54 in the daisy chain of memorychips 47, but has a plurality of data bus 59 chains, none of whichinterconnect the entire daisy chain of memory chips 47.

Memory system 270 shown in FIG. 12 is similar to memory system 270 shownin FIG. 8 or FIG. 9A in the sense that both have a chained bus clock(and timing signals 62 when memory chips 54 are not self timed) and thataddress/command bus 58 runs through a daisy chain of memory chips 47 oncarrier 56; four memory chips 54 (54A, 54B, 54C, 54D) are shown.Address/command bus 58A is driven by memory controller 52;address/command bus 58B is driven by memory chip 54A; address/commandbus 58C is driven by memory chip 54B; address/command bus 58D is drivenby memory chip 54C; and address/command bus 58E is driven by memory chip54D. Address/command bus 58E is shown “floating”; in another embodiment,it can be connected back to memory controller 52 in a similar manner asaddress/command bus 58N in FIG. 8. FIG. 12 shows a first data bus chainhaving data bus 59A1, 59B1, and 59B2. A second data bus chain has databus 59C1, 59D1, and 59D2.

In an embodiment, each data bus chain couples a single memory chip 54 tomemory controller 52 so that data words 130 do not flow through memorychips 54 that are not target chips for the packets as identified by thepacket ID 132 (FIG. 7B, 7C) or chip ID 133.

FIG. 13 is a block diagram of memory controller 52. It is understoodthat there are many, often complicated, logic blocks implemented invarious memory controllers. The present example memory controller 52shown in FIG. 13 is suitable to explain embodiments of the invention.

Bus 223 (first shown in FIG. 2) couples memory controller 52 toprocessor 200. Processor bus 223 carries transmissions of requests forreads and writes to memory 210, along with data to be written intomemory 210. Processor bus 223 also carries transmission of data that hadbeen requested by processor 200 from memory controller 52 to processor200. Typically, addresses transmitted on bus 223 are logical addresses.Processor bus control 670 receives the requests for reads and writestransmitted on bus 223 as well as data that is to be written to memory210, and places such requests in processor request queue 605.

Logical/physical translate 610 translates the logical address receivedfrom the processor and, with information of the physical memoryresources available (e.g., number of daisy chains of memory chips 47,number of carriers 56, amount of storage in each memory chip 54)translates the logical address into a physical address.

Create chip ID 620 uses the physical address and information as to thephysical memory resources available to determine which daisy chain ofmemory chips 47 contains the physical address, and to create a chip ID(e.g., chip ID 121 shown in FIGS. 5A, 5B). For example, memorycontroller 52 has information regarding how many daisy chains of memorychips 47 are attached to memory controller 52, as well as amount ofstorage in each memory chip 54, and amount of data read or written toeach memory chip 54 at one time. Memory controller 52 maintains amapping (not shown) between each physical address (or, alternatively,address range) and which memory chip 54 holds data for that address oraddress range.

In an embodiment, the chip ID 121 is simply one or more bits fromaddress 124. That is, the mapping mentioned above is a portion ofaddress 124. In an embodiment in which chip ID is simply a portion ofaddress 124, create chip ID 620 is not needed, as logical/physicaltranslate 610 produces a physical address in such an embodiment thatincludes the portion of address 124 that identifies which chip anaddress/command word 120 is directed to.

Packet manager 630 assigns and keeps track of packets sent on each daisychain of memory chips 47. For example, if there are 64 daisy chains ofmemory chips 47, and the memory controller supports sixteen outstandingrequests on each daisy chain of memory chips 47, packet manager 630 mustassign a unique packet ID 123 (such as zero to fifteen) for eachoutstanding address command word 120 on each daisy chain of memory chips54. When a particular address/command word 120 is satisfied (i.e., theread or write is completed successfully) the corresponding packet ID 123can be used again. Address/command generator 640 uses requestinformation for a particular request in processor request queue 605, theaddress determined in logical physical translate 610, the chip ID anddaisy chain of memory chips 47 determined in create chip ID 620, and thepacket ID value determined in packet manager 630 to create anaddress/command word 120. The address/command word 120 is transmitted onan address/command bus 58 according to the daisy chain of memory chips47 determined in create chip ID 620. Address/command busses 58 are shownas 58-1 to 58-N. Packet manager 630 is not required in some embodimentsas explained in reference to FIG. 18C, in which a sequencer 621 ensuresin-order return of data from memory chips 54.

It will be understood that the same address/command word 120 can be sentto a plurality of daisy chains of memory chips 47, in order to quicklyaccess a large amount of data in parallel. For example, if each memorychip 54 returns 64 bits in data word 130 (i.e., eight bytes), parallelaccess over eight daisy chains of memory chips 47 will return 64 bytes.

Data send/receive 650 sends data words for writes, using packet IDsassigned by packet manager 630, along with data from an associatedprocessor write request in processor request queue 605. Data words 130are sent to the proper daisy chain of memory chips 47 over data busses59 (59-1 to 59-M shown). Data send/receive 650 also receives data comingback on data busses 59 and uses the packet ID 132 in the incoming readdata word 130 to associate data 131 with the proper processor request.Data from read data word 130 is placed in processor response queue 660.Processor bus control 670 takes data 130 from processor response queue660 and transmits data from read data word 130 back to processor 200.

In the examples given, memory chips 54, for exemplary purposes, havebeen described in terms of DRAM (dynamic random access memory) memorychips. However, it will be understood that memory chips 54, inembodiments, are SRAM (static random access memory) memory chips. SRAMchips typically also have circuitry that precharges bit lines and memorycells in such SRAM chips discharge either a true or a complementary bitline coupled to the memory cells. Self timed SRAM memory chips are alsocontemplated in embodiments of the invention. Such self timed SRAMmemory chips, in embodiments, are daisy chained as shown as memory chips54 in FIG. 9A, which uses bus clock 60 to provide address/command bus 58and data bus 59 timings, while arrays 55 in memory chips 54 are accessedin a self-timed manner as explained earlier similar to when memory chips54 are implemented in DRAM technology. Embodiments of the inventionfurther contemplate daisy chains of memory chips 47 in which some of thememory chips 54 are SRAM memory chips and some of the memory chips 54are DRAM memory chips.

Embodiments of the invention are also expressed as methods. FIG. 14 is ahigh level flow chart of a method embodiment 700 of the invention.Method 700 begins at step 702. In step 704, a processor, such asprocessor 200 (FIG. 2) sends a request for a read or a write to a memorysystem such as memory system 270 in FIG. 2. The memory system furthercomprises a memory (memory 210) which further comprises one or moredaisy chains of memory chips, the memory chips connected to the memorycontroller by a bus clock chain, an address/command bus chain, and adata bus chain.

In step 706, the memory controller sends a bus clock down a bus clockchain. The bus clock is received by a first memory chip in the daisychain of memory chips, and is re-driven by the first memory chip to asecond memory chip, and so on, through the daisy chain of memory chips.The bus clock is used to control how fast (i.e., at what frequency)address/command words and data words are transmitted on anaddress/command bus and on a data bus.

In step 708, if memory chips in the daisy chain of memory chips are notself timed, the memory controller transmits timing signals to a chain oftiming signals. The timing signals are used by the memory chips to timeaccesses to arrays on the memory chips. FIG. 4 shows a timing block 63that receives one link of the timing signal chain and drives a secondlink of the timing signal chain.

In step 710, the memory controller produces an address/command word,such as is shown in FIGS. 5A and 5B, and transmits the address/commandword into a chain of address/command busses. FIG. 4 shows a memory chip54 that receives a first link (58X) of the address/command bus and isconfigured to re-drive an address/command word on a second link of theaddress command bus (58Y).

In step 712, the memory controller produces a data word that isassociated with the address command word when the address/command wordis for a write. The data word is transmitted into a chain of databusses. FIG. 4 shows a memory chip 54 that receives a first link (59X)of the data bus and the memory chip is configured to re-transmit thedata word onto a second link (59Y) in the chain of data busses.

in step 716, if the address/command word is for a read, the memorycontroller receives a read data word from the data bus that isassociated with the request for data.

In step 718, the memory controller transmits the requested data to theprocessor. Step 720 ends the method.

FIG. 15 is a high level flow chart of method 750 used by a memory chipin a daisy chain of memory chips. Step 752 begins the process. In step754, the memory chip receives a bus clock from a first link in a busclock chain (see FIG. 4, bus clock 60X). If the memory chip is not thelast memory chip in a chain of memory chips, the memory chip re-drivesthe bus clock onto a second link in the bus clock chain (see FIG. 4, busclock 60Y). The bus clock is used to determine a frequency thataddress/command words are transmitted/received on an address/command busand a frequency that data words are transmitted/received on a data bus.

In step 756, the memory chip services address/command bus activity. Step756 is described in more detail in FIG. 16. In step 758, the memory chipprocesses data bus activity. As described supra in detail in theapparatus description, In brief, data words arriving on the memorycontroller side of the memory chip (FIG. 4, data bus 59X_(A)) areexamined for, in various embodiments, a chip ID in the data word thatmatches the memory chip ID of the present memory chip, or a packet ID ina data word that matches a packet ID in an address/command word. If amatch occurs, data in the data word is stored in an array in the presentmemory chip. If not, the data word is re-transmitted to the next memorychip in the daisy chain of memory chips (FIG. 4, data bus 59Y_(A)). If adata word is received on a distal side of the memory chip from thememory controller (FIG. 4, data bus 59Y_(B)), the data word isre-transmitted by the memory chip onto a data bus link on a data buslink on a proximal side, relative to the memory controller, of thememory chip (FIG. 4, data bus 59X_(B)). Step 760 ends method 750.

FIG. 16 shows a more detailed flow chart of step 756. Step 780 startsmethod 756. Step 782 receives an address/command word on a point topoint link of the address/command bus chain (FIG. 4, address/command bus58X).

Step 784 checks to see if the present address/command word is for thepresent memory chip. In an embodiment explained supra with reference toFIG. 6A in which the address command bus is one bit wide, if the firstbit received is a “1”, the address/command word is for the presentmemory chip. In embodiments in which the address/command bus is morethan one bit wide, the chip ID is checked against a memory chip ID withvarious implementations of a logical comparison (FIG. 6D, chip IDcompare 87).

If the address/command word is not for the present memory chip, controlpasses to step 786, which re-drives the address/command word on a pointto point interconnection link (FIG. 4, address/command bus 58Y) to anext chip in the daisy chain of memory chips. In the embodimentdescribed with reference to FIG. 6A, the chip ID field of theaddress/command word is shifted right by one bit position.

If step 784 determines that the address/command word is directed to thepresent chip, control passes to step 788. Step 788 looks at a commandfield in the address/command word and determines if the address/commandword is for a read or a write. If the address/command word is for aread, control passes to step 790. Step 790 uses address information inthe address/command word to make a read access to an array on the memorychip. In an embodiment explained earlier, array timings are self timed,so that a relatively faster chip will be accessed faster than arelatively slow chip. In another embodiment also explained earlier,memory timings are transmitted on a timing signal chain through thedaisy chain of memory chips.

In step 794 read data is associated with a packet ID of theaddress/command word to make a data word that can be transmitted back tothe memory controller.

In step 796 the read data word is placed in a read data buffer and istransmitted on a data bus point to point link in the direction of thememory controller when the data bus link is available.

Steps 794 and 796 are not required in embodiments of the invention wherethe memory controller manages transmission of address/command words insuch a way as to guarantee in-order return of data words.

If step 788 determines that the address/command word is a write dataword, data is written into an array on the memory chip. In anembodiment, the data written into the array was transmitted as part ofthe address/command word (See FIG. 5B). In another embodiment, the datawas transmitted down the data bus chain by the memory controller and wasassociated on the present chip (e.g., by a chip ID in the write dataword, or by matching a packet ID in the write data word) with theassociated address/command word).

Method 800 is depicted in FIG. 17. Method 800, in brief, provides forself timing of an array on a memory chip. Self timing eliminates a needto provide external control (e.g., from a memory controller) of arraytiming on a memory chip. Self timing also allows each memory chip in thedaisy chain of memory chips to access an array at a speed appropriatefor the array on each memory chip. Step 802 begins the method.

Step 804 dynamically determines permissible access timing of an array ona memory chip. Access timings of an array are determined by processvariations in a semiconductor process that produced the memory chip;current voltage conditions; and current temperature of the memory chip.FIGS. 9B, 10, 11A, and 11B provide apparatus in detail sufficient forone of ordinary skill in the art to construct a self time circuit thatproduces self timing for the array. The step includes running a ringoscillator having a frequency determined by a dynamic circuit thattracks actual access times of the array. Step 806 comprises using thering oscillator frequency to produce timing signals and controlling thearray with those timing signals.

Embodiments of the invention explained above provide a very highperforming memory system 270. Other embodiments are possible that, whileperforming at a lower speed, require less logic on memory chips 54. Inbrief, memory controller 52, in such embodiments, is responsible forensuring that data is received in order and that no “collisions” on anydata bus 59 link can occur. That is, when a first read request is sentto a memory chip 54 on a first address/command word 120, and a secondread request is sent to a memory chip 54 on a second address/commandword 120, a first data word 130 that corresponds to the firstaddress/command word 120 is guaranteed to arrive at memory controller 52before a second data word 130 that corresponds to the secondaddress/command word 120 arrives at memory controller 52. To accomplishthis, memory controller must know how long it takes for anaddress/command word 120 to propagate down the address/command bus 58chain, and access times for each memory chip 54 in the daisy chain ofmemory chips 47.

Embodiments of an address/command word 120, a data word 130, and amemory controller 52 suitable for an embodiment of the inventionsuitable for a memory system 270 in which all timings are enforced bymemory controller 52 is shown in FIGS. 18A, 18B, and 18C. Suchembodiments will be called “simplified memory chip embodiments”.

FIG. 18A shows an address/command word 120, suitable for a simplifiedmemory chip embodiment, having command 122 as the first field, that is,the first information from address/command word 120 sent by memorycontroller 52. Address 125 comprises a physical address, a portion ofwhich (i.e., a number of high order bits in the physical address)determines which memory chip 54 in a daisy chain of memory chips 47 theaddress/command word 120 is directed to. In the embodiment of FIG. 18A,data 131 to be written to a memory chip 54 is sent as part ofaddress/command word 120 as explained earlier. A CRC 125 field isincluded in embodiments requiring the additional data integrity offeredby CRC data. Command 122 as the first information received by eachmemory chip 54 in a daisy chain of memory chips 47 tells the each memorychip 54 if data 131 is a portion of a particular address/command word120 (i.e., if command 122 is for a “write”) or if data 131 is not aportion of a particular address/command word 120 (i.e., if command 122is for a “read”). No packet ID is required because, in the simplifiedmemory chip embodiment being discussed, memory controller 52 enforcestimings such that there is no ambiguity regarding an address/commandword 120 and a data word 130.

FIG. 18B shows a data word 130 appropriate for the simplified memorychip embodiment being discussed. As above, no packet ID 132 is required.Only data 131 and CRC 135 (in embodiments implementing CRC) are used.

FIG. 18C shows a memory controller 52 that enforces timings such that noambiguities exist between an address/command word 120 and a data word130. Like referenced items function as explained in reference to FIG.13. Sequencer 612 knows how fast address/command words 120 aretransmitted and how fast data words 130 are transmitted. Sequencer 621knows how many signal conductors in address/command bus 58 (e.g.,scanned in at startup of computer system 250); and a length of anaddress/command word 120 (e.g., scanned in at startup of computer system250), and therefore knows how many cycles are required to send anaddress/command word 120. Sequencer 621 also knows how long each memorychip 54 in a daisy chain of memory chips takes to re-drive anaddress/command word 120. Similarly, sequencer 621 knows how many signalconductors are in data bus 59, and various timing requirementsassociated with data bus 59, such as time to receive data, time tore-drive data. Sequencer 621 also knows how long a memory chip 54 takesto access (i.e., read or write) an array on a memory chip 54. Sequencer621 controls address/command generator 640 to transmit address/commandwords 120 at times that guarantee that reads will be returned in theproper order (i.e., first read data words 130 will arrive in the orderof address/command words 120.

In the simplified memory chip embodiment being discussed, (See FIG. 6A)address/command queue 81 in address/command 80 on each memory chip issimply a single address/command buffer 82. Referring to FIG. 7A asimplemented for a simplified memory chip embodiment, Read queue 141 is asimple buffer (register) and add packet 142 is not needed. Write queue145 is a simple buffer. Write queue 146 is not needed, since data 131 issent with address/command words 120. Read queue 147 is not needed, sincedata logic 100 simply re-drives data words 130 from data bus link 59Y todata bus link 59X. Data bus 59 is assumed to be unidirectional, in thedirection of memory controller 52, so no apportioning of data bus 59into an “incoming portion” and an “outgoing portion” is needed.

FIG. 6B will be used to illustrate how sequencer 621 operates. Assumenow that each address/command bus 58 link has four signal conductors. Itwill be understood that, if “single ended” transmission is implemented,a signal conductor is just one “wire”; if differential transmission isimplemented, a signal conductor has two wires. The following assumptionsare shown in Table 1, for exemplary purposes:

TABLE 1 Assumption Description 0.2 ns Address/command bus beat 0.2 nsMemory chip receive 0.2 ns Memory chip time to recognize chip ID Addressportion 0.2 ns Memory chip re-drive on address/command bus 30 ns Accesstime on memory chip 0.2 ns Data bus beat 0.2 ns Memory chip receive timefrom data bus 0.2 ns Memory chip re-drive on data bus 24 bits Length ofAddress/command word for read 4 bits Number of signal conductors inaddress/command bus 64 bits Data word length 8 bits Number of signalconductors in data bus 31.6-33.2 ns First & last data transfer fromfirst chip from a/c word 32.2-33.8 ns First & last data transfer fromsecond chip from a/c word 32.8-34.4 ns First & last data transfer fromthird chip from a/c word 33.4-35.0 ns First & last data transfer fromfourth chip from a/c word

TABLE 2 Last Access to Read from Chip 1 Chip 2 Chip 3 Chip 4 Chip 1 30ns 2.6 ns 3.6 ns 4.6 ns Chip 2 1.2 ns 30 ns 2.6 ns 3.6 ns Chip 3 −0.2 ns0.8 ns 30 ns 0.8 ns Chip 4 −1.2 ns −0.2 ns 0.8 ns 30 ns

The timings of first and last data transfer from memory chips 1-4 shownin Table 1 are measured from transmission by memory controller 52 of anaddress/command word 120 having a read command.

Table 2 gives required timings for issuance of an address/command word120 by memory controller 52 (i.e., timed by sequencer 621 in FIG. 18C).For example, accesses to a particular chip (in the simplified memorychip embodiment being discussed now) cannot occur faster than 30 ns,because the access time of each chip is assumed to be 30 ns, and thereis no queuing in memory chips 54 in the simplified memory chipembodiment. However, if an access were made to chip 1 at 0.0 ns (i.e., afirst memory chip 54 in the daisy chain of memory chips 47) anaddress/command word 120 can be launched by memory controller 52 at 1.2ns. Note that, in the example, that if a first data word 130 is to beread from chip 1, and a subsequent data word is to be read from chip 4,there is a −1.2 ns timing requirement. Note that transmission of a24-bit data word 120 requires six beats on the address/command bus 58 at0.2 ns/beat in the example, or 1.2 ns to transmit an address/commandword 120. Sequencer 121 therefore can not transmit the address/commandword 120 to the fourth chip out of sequence before transmitting theaddress/command word 120 to the first chip, as a collision would occurbetween a first beat of data transmission from the first memory chip 54and a last beat of data transmission from the fourth memory chip 54.However, if (using the same assumptions) the daisy chain of memory chips47 had more memory chips, sequencer 621 could transmit address/commandwords 120 out of order in some cases, with data words 130 arriving inorder. FIG. 20 illustrates method 2000, in flow chart form, data words130 being received in a different order than address/command words 120,as described above. Method 2000 begins at block 2002. Blocks 2002-2012are performed in sequence. In block 2004 a first address/command word istransmitted by sequencer 621 in a memory controller to read a first dataword. In block 2006, a second address/command word is transmitted toread a second data word. In block 2008, the second data word is receivedin the memory controller. In block 2010, the first data word is receivedin the memory controller. block 2012 ends method 2000.

It will be understood that the above example is simplified forexplanation. Further timing requirements by sequencer 621, such as readsafter writes (note that write address/command words have a data 131field, for example) must be accommodated. In general, read access andwrite access times are not the same for a particular memory chip 54. Theexample is used to illustrate that memory chip 52, in an embodiment, iscapable of issuing address/command words timed to avoid collisions ondata bus 130.

The simplified memory chip embodiment being discussed is also applicableto a daisy chain of memory chips 47 in which the memory chips 54 areself timed as described earlier. However, sequencer 621 must have datafrom each self timed memory chip 54 regarding current access timing ofthe arrays 55 on each self timed memory chip 54. In an embodiment,memory controller 52 transmits, on a periodic basis (a calibration ofaccess timing information), an address/command word 120 having a command(command 122, shown in FIGS. 5A, 5B) that causes the memory chip 54 towhich the address/command word 120 is directed, to create one or moredata words 130 having access timing information, and to transmit the oneor more data words 130 having the access timing information to memorycontroller 52. In an embodiment, address/command words 120 for reads andwrites are suspended during the calibration of access timinginformation. Sequencer 621 uses the access timing information to adjustwhen address/command words 120 are transmitted.

In an alternative embodiment, memory controller 52 suspends transmissionof address/command words 120 resultant from requests for reads andwrites from processor 200. Memory controller 52 creates a particularaddress/command word 120 with a calibrate command in command 122 andtransmits the particular address/command word to the daisy chain ofmemory chips, receives a responsive data word 130, storing the time fromtransmission of the particular address/command word 120 to the receptionof the responsive data word 130. Memory controller 52 uses information(e.g., chip ID) as to which memory chip 54 in the daisy chain of memorychips 47 was addressed. Memory controller 52 also has information as tolengths of address/command word 120 and data word 130 and frequency andwidth of address/control bus 58 and data bus 59. From this information,memory controller can compute the array access time (for example, arrayaccess time is the total time minus total transmission time). Arrayaccess time may be different for a read versus a write. Self timedmemory chip 54, on a write, must transmit a data word 130 back to memorycontroller 52 to acknowledge completion of a write access to array 55 onthe self timed memory chip 54.

1. A memory system comprising: a memory controller; a memory comprisinga first memory chip and a second memory chip; a first segmentedunidirectional data bus for transmitting write data words, the firstsegmented data bus further comprising: a first segment connected to thememory controller and to the first memory chip; and a second segmentconnected to the first memory chip and to the second memory chip; asecond segmented unidirectional data bus for transmitting read datawords, the second segmented unidirectional data bus further comprising:a third segment connected to the memory controller and to the firstmemory chip; a fourth segment connected to the first memory chip and tothe second memory chip; wherein a first read data word is received bythe first memory chip from the fourth segment and stored on the firstmemory chip in a read data queue until the first memory chip hascompleted transmitting a second read data word on the third segment; thefirst read data word being transmitted from the read data queue onto thethird segment when transmission of the second read data word on thethird segment has been completed; and wherein the first read data wordis associated with a third address/command word, and the second readdata word is associated with a fourth address/command word; wherein afirst write data word is received by the first memory chip from thefirst segment and stored in a queue on the first memory chip until thefirst memory chip has completed transmitting a second write data word onthe second segment, and wherein the first write data word is associatedwith a first address/command word and the second write data word isassociated with a second address/command word; and wherein the firstwrite data word is transmitted from the queue onto the second segmentonly if the first write data word is to be written to the second memorychip; wherein the third address/command word was issued by the memorycontroller prior to the fourth address/command word; and wherein eachdata word is identifiable with an ID (identifier) which is transmittedwith the each data word, the ID usable by the memory controller tore-order the data words.